Skip to content

fix(context): inject project dream namespaces alongside recent raw memory (closes #2834)#2858

Open
rodboev wants to merge 9 commits into
thedotmack:mainfrom
rodboev:fix/2834-dream-context-injection
Open

fix(context): inject project dream namespaces alongside recent raw memory (closes #2834)#2858
rodboev wants to merge 9 commits into
thedotmack:mainfrom
rodboev:fix/2834-dream-context-injection

Conversation

@rodboev

@rodboev rodboev commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Distilled memories stored under <project>:dream were absent from normal project context injection because the default query path only used the cwd-derived raw project name. This PR adds the dream namespace to the default project list and keeps a bounded recent raw-memory fallback so distilled context becomes primary without hiding fresh undistilled observations.

Why

generateContext() in src/services/context/ContextBuilder.ts:105-128 already supports multi-project queries, and queryObservationsMulti() / querySummariesMulti() in src/services/context/ObservationCompiler.ts:88-175 already union rows across project lists. The gap was that getProjectContext() in src/utils/project-name.ts:76-96 never emitted the companion dream namespace for ordinary project sessions, so the multi-project path was not used for distilled memory at all.

Scope

This PR is intentionally read-path only. It does not change how dream rows are written, does not migrate stored project names, and does not inject any unscoped global dream namespace.

Risk

The main risk is over-weighting old distilled rows and crowding out fresh raw context. The fix keeps that bounded by preserving a recent raw fallback and by leaving the rendered project label unchanged even though the underlying query list expands.

Verification / Test plan

  • bun test tests/context/observation-compiler.test.ts - 9 passed; includes the raw-row fallback when dream rows would otherwise saturate the selected result set.
  • npm run build - passed locally and regenerated bundled context/worker artifacts.
  • npm run lint:hook-io - passed locally.
  • npm run lint:spawn-env - passed locally.
  • bun test tests/utils/project-name.test.ts - not rerun as a full file; a previous full-file run had pre-existing Windows ~ expectation failures.
  • npm run strip-comments:check - failed against the existing repo-wide comment-stripping baseline in check mode (Changed: 327), unrelated to this dream-namespace branch.

Closes #2834

@greptile-apps

greptile-apps Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes distilled (dream) memories being absent from normal project context injection by wiring the companion :dream namespace into getProjectContext's allProjects list and routing generateContextWithStats through the existing multi-project query path when dream rows exist. A raw-row fallback (includeRawFallback) is added so fresh undigested observations are never fully displaced by older distilled ones.

  • project-name.ts: getProjectContext now calls withDreamProject(...) for both normal and worktree sessions, emitting ['proj:dream', 'proj'] (or the worktree variant with four entries) as allProjects.
  • ContextBuilder.ts: Added useDreamQueries gate that counts dream-namespace rows before enabling the multi-query path; new getPrimaryContextProject helper strips dream entries so the rendered project label is unchanged.
  • ObservationCompiler.ts: Added includeRawFallback and queryLatestRawObservation to guarantee at least one raw observation in the result when dream rows would otherwise saturate the limit; queryObservations (single-project) now also selects o.project so getPriorSessionMessages can correctly skip dream-project rows.

Confidence Score: 5/5

Safe to merge; the change is read-path only and degrades gracefully when dream rows are absent or sparse.

The core logic — wiring the dream namespace into allProjects, gating on countObservationsByProjects before enabling the multi-query path, and the raw-row fallback — is straightforward and well-covered by the new tests. queryObservations now correctly selects o.project so getPriorSessionMessages can filter dream rows, and the include-last-message-dot-path test verifies the end-to-end behaviour with a dream row present. The two observations raised are minor: a null-project row could trigger an unnecessary extra fallback DB call, and the telemetry search_strategy field does not distinguish the new multi-query path — neither affects correctness or user-facing behaviour.

No files require special attention; ObservationCompiler.ts has one minor defensive-null gap in includeRawFallback worth a one-character fix.

Important Files Changed

Filename Overview
src/utils/project-name.ts Adds getDreamProjectName export and withDreamProject helper; getProjectContext now always emits dream namespaces in allProjects (except for null-cwd fallback). Logic is clean and covered by tests.
src/services/context/ObservationCompiler.ts Adds includeRawFallback/queryLatestRawObservation for raw-row guarantee; queryObservations now selects o.project; getPriorSessionMessages correctly skips dream rows. One subtlety: row.project treats null rows the same as dream rows, potentially triggering an extra fallback query unnecessarily.
src/services/context/ContextBuilder.ts New useDreamQueries gate, getPrimaryContextProject, and multi-query routing; logic is clean and well-tested. Telemetry search_strategy still only distinguishes full vs timeline, not single vs multi query.
tests/context/observation-compiler.test.ts New tests cover dream-row skipping in getPriorSessionMessages and the raw-fallback saturation scenario in queryObservationsMulti; assertions are specific and correct.
tests/utils/project-name.test.ts New assertions on allProjects for the normal-path and worktree cases confirm dream namespaces are injected correctly; null-cwd allProjects invariant still lacks coverage.
tests/context/context-builder.test.ts Adds getPrimaryContextProject unit tests and a dream-direct-query integration test; mock for countObservationsByProjects always returns 1, so the no-dream-rows fallback path is untested.
tests/context/include-last-message-dot-path.test.ts Extended to include a dream row as the first observation, confirming getPriorSessionMessages correctly skips it and finds the raw-session transcript.
src/services/telemetry/telemetry.ts Adds __resetTelemetryStateForTesting alias for backward-compatible test reset; no logic changes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["generateContextWithStats(input)"] --> B["getProjectContext(cwd)\nallProjects = [proj:dream, proj]"]
    B --> C{input.projects\nsupplied?}
    C -- yes --> D[use input.projects]
    C -- no --> E[use context.allProjects]
    D & E --> F["split into\ndreamProjects / rawProjects"]
    F --> G{dreamProjects exist\nAND count > 0?}
    G -- no --> H["queryProjects = rawProjects"]
    G -- yes --> I["queryProjects = all projects\n(dream + raw)"]
    H & I --> J{queryProjects.length > 1?}
    J -- no --> K["queryObservations(single)"]
    J -- yes --> L["queryObservationsMulti(multi)"]
    L --> M["includeRawFallback()"]
    M --> N{all selected rows\nare dream rows?}
    N -- no --> O[return as-is]
    N -- yes --> P["queryLatestRawObservation()\nreplace last dream row"]
    P --> O
    K & O --> Q["getPriorSessionMessages\nskips dream rows via isDreamProject()"]
    Q --> R["buildContextOutput\nproject label = getPrimaryContextProject\n(last raw project)"]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["generateContextWithStats(input)"] --> B["getProjectContext(cwd)\nallProjects = [proj:dream, proj]"]
    B --> C{input.projects\nsupplied?}
    C -- yes --> D[use input.projects]
    C -- no --> E[use context.allProjects]
    D & E --> F["split into\ndreamProjects / rawProjects"]
    F --> G{dreamProjects exist\nAND count > 0?}
    G -- no --> H["queryProjects = rawProjects"]
    G -- yes --> I["queryProjects = all projects\n(dream + raw)"]
    H & I --> J{queryProjects.length > 1?}
    J -- no --> K["queryObservations(single)"]
    J -- yes --> L["queryObservationsMulti(multi)"]
    L --> M["includeRawFallback()"]
    M --> N{all selected rows\nare dream rows?}
    N -- no --> O[return as-is]
    N -- yes --> P["queryLatestRawObservation()\nreplace last dream row"]
    P --> O
    K & O --> Q["getPriorSessionMessages\nskips dream rows via isDreamProject()"]
    Q --> R["buildContextOutput\nproject label = getPrimaryContextProject\n(last raw project)"]
Loading
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
src/services/context/ObservationCompiler.ts:86
The guard uses `row.project && !isDreamProject(row.project)`, which is falsy when `row.project` is `null`. Any observation row with a null `project` column (possible for data written before the column was populated) is treated the same as a dream row: the `selected.some(...)` check fails to recognise it as a raw row, so `queryLatestRawObservation` is invoked unnecessarily. The result set is still correct, but you pay an extra DB round-trip on every context injection for projects that have legacy null-project rows.

```suggestion
  if (selected.some(row => row.project != null && !isDreamProject(row.project))) return selected;
```

### Issue 2 of 2
src/services/context/ContextBuilder.ts:165
The `search_strategy` telemetry field only captures `'full'` vs `'timeline'`, but this PR introduced a third dimension: single-project vs multi-project query. A user who never uses `input.full` but whose dream namespace is populated will always hit `queryObservationsMulti`, and that change in query shape is invisible to the telemetry. Adding a `'multi'` variant (or a separate boolean field) would make it possible to track how often the new dream-augmented path is exercised in production.

```suggestion
    search_strategy: full ? 'full' : 'timeline',
    // TODO: expose useMultiQuery here once the stat shape is stable
```

Reviews (18): Last reviewed commit: "fix(telemetry): keep shared telemetry te..." | Re-trigger Greptile

Comment thread src/services/context/ObservationCompiler.ts Outdated
@rodboev

rodboev commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Added the worktree composite dream namespace to allProjects, enforced a raw-row fallback slot when dream rows fill the limit, and covered both behaviors in focused tests. The previous red CI on this branch matched the shared baseline now isolated in #2853, this push is the branch-local Greptile follow-up.

@thedotmack

Copy link
Copy Markdown
Owner

Review finding that blocks this one: prioritizeProjectRows breaks recency-ordering assumptions downstream. It returns [dream-rows..., raw-rows...], but three consumers assume index 0 is the most recent row: getPriorSessionMessages() takes element 0's session as "the prior session" (would resolve to an old dream-distillation session), prepareSummariesForTimeline() uses allSummaries[i+1] as the older summary for displayEpoch, and buildContextOutput's summaries[0]/observations[0] semantics shift. buildTimeline re-sorts, but those three don't. Second concern: nothing in the tree writes a :dream project namespace (zero grep hits across src/, openclaw/, scripts/), so this adds a read path — plus an inflated SQL LIMIT on every normal session — for a producer that doesn't exist in-repo yet.\n\nIf the dream namespace is coming from an external workflow, suggest: interleave by epoch instead of prepending (preserve sort order), and gate the multi-project query path behind the namespace actually having rows. Happy to merge a revision.

@rodboev rodboev force-pushed the fix/2834-dream-context-injection branch 2 times, most recently from 96ae63d to 91a88e2 Compare June 10, 2026 10:33
@rodboev

rodboev commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in the latest push (9d05ef87). prioritizeProjectRows is gone. The dream/raw split now happens at the SQL query level: ContextBuilder.generateContextWithStats() checks whether dream rows actually exist via countObservationsByProjects/countSummariesByProjects before including dream projects in the query set. When dream rows exist, the queries use queryObservationsMulti/querySummariesMulti which union the projects in a single ORDER BY created_at_epoch DESC query, so recency ordering is preserved end-to-end rather than prepended.

getPriorSessionMessages already filters out dream-project observations (line 271 of ObservationCompiler.ts), prepareSummariesForTimeline and buildContextOutput both receive the SQL-sorted results directly, and summaries[0] remains the most recent row regardless of namespace. The inflated LIMIT only activates when dream rows exist, gated by the useDreamQueries check.

@rodboev rodboev force-pushed the fix/2834-dream-context-injection branch from 13f6401 to c1c8db0 Compare June 19, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix distilled memory context injection for project dream namespaces

2 participants